Prime end rotation numbers of invariant separating continua 
of annular homeomorphisms 

Shigenori Matsumoto 

Abstract. Let / be a homeomorphism of the closed annulus A isotopic to 
the identity, and let X C IntA be an /-invariant continuum which separates A 
into two domains, the upper domain U+ and the lower domain U—. Fixing a 
lift of / to the universal cover of A, one defines the rotation set p{X) of X by 
means of the invariant probabilities on X, as well as the prime end rotation 
number p± oi U±. The purpose of this paper is to show that p± belongs to 
p{X) for any separating invariant continuum X. 



1. Introduction 

Let / be a homeomorphism of the closed annulus A ~ x [—1, 1], isotopic to 
the identity, i. e. / preserves the orientation and each of the boundary components 
d±A = X {±1}. Suppose there is an /-invariant partition oi A; A = C/_ UXUU+, 
where U± is a connected open set containing the boundary component d±A and X 
is a connected compact set. Let 

7r:i = Rx [-1,1] ^S'l x [-1,1] 

be the universal covering map and T : A ^ A a, generator of the covering transfor- 
mation group; T{^,ri) — + 1,?/). Denote by p : A — > M the projection onto the 
first factor. 

Fix once and for all a lift f : A ^ A oi f. Then the function p o f — p 
is T-invariant and can be looked upon as a function on the annulus A. Define 
the rotation set p{X) as the set of values ^(p ° f ~ p), where ^ ranges over the 
/-invariant probability measures supported on X. The rotation set is a compact 
interval (maybe one point) in R, which depends upon the choice of the lift / of /. 

The first example of an invariant continuum X such that the frontiers of U± 
satisfy Fr(J7+) = Fr(?7_) = X and that the rotation set p{X) is not a singleton is 
constructed by G. D. Birkhoff in his 1932 year paper [Bj, and is refered to as a 
Birkhoff attractor. It turns out that the Birkhoff attractor is an indecomposable 
continuum ([Cl IL2j ). Furthermore it is shown by P. Le Calvez ([LI]) that for 
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any rational number between the two prime end rotation numbers is realized by a 
correspoding periodic point of /. 

Let U± = U±LidooU± be the prime end compactification of U±, where dooU± is 
the space of the prime ends ([El [Ml IMNj ). The space dooU± is homeomorphic to 
the circle and U± to the closed annulus. As is well known, the homeomorphism / 
restricted to U± extends to a homeomorphism f± : U± U±. Denoting /+ ~ [0, 1] 
and /_ = [—1,0], define a homeomorphism 

*± : J7± ^ 5^ X /± 

such that 'if ±{dooU±) — xO. By some abuse of notations denote by tt : U± U± 
the universal covering map. Thus 7t^^{U±) is considered to be a subspace of both 
A and U±. Let f±:jJ±^ C7± be the lift of /± such that /± = / on ti~^{U±). The 
rotation number of the restriction of f± to 7r~^((9ooC^±), denoted by p±, is called 
the prime end rotation number of U± . 

The purpose of this paper is to show the following. 

Theorem 1. The prime end rotation number p± belongs to p{X). 

This result is aheady known for X = Fr(f/_) = Fr([/+) ([BG]), and for any X 
if the homeomorphism / is area preserving (Lemma 5.4, [FLj ). 

It is shown in Theorem 2.2 of [F that any rational number in p{X) is realized 
by a periodic point if X consists of nonwandering points. Notice that then X, 
consisting of chain recurrent points, is chain transitive since it is connected, and 
thus satisfies the condition of Theorem 2.2. As a corollary we have 

Corollary 2. If X consists of nonwandering points and if p/q lies in the closed 
interval bounded by and p^, then there is a point x £ Tr^^(X) such that f^ix) — 
TP{x). 

In what follows we also use the following notation. Let 

*± : t/± ^ M X /± 

be a lift of and define p± : U± — > M by p± = p o The projection p± is 
within a bounded error of p on 7r~^(C) for a compact domain C of U±. But they 
may be quite different on the whole 7r~^([/±). 

2. Proof 

First of all let us state a deep and quite useful theorem of P. Le Calvez ( |L3j ) 
which plays a key role in the proof. A fixed point free and orientation preserving 
homeomorphism F of the plane is called a Brouwer homeomorphism. A proper 
oriented simple curve 7 : M — )■ is called a Brouwer line for F if ^"(7) C Rij) and 
^"""^(7) C ^(7), where R{'j) (resp. ^(7)) is the right (resp. left) side complementary 
domain of 7, which is decided by the orientation of 7. 

Theorem 2.1. Let F be a Brouwer homeomorphism commuting with the elements 
of a group T which acts on freely and properly dis continuously. Then there is 
a T -invariant oriented topological foliation o/R^ whose leaves are Brouwer lines of 
F. 

The proof of Theorem [Tj is by absurdity. Assume in way of contradiction that 
P- < p/q < mip{X). Considerng f^T^P instead of /, it suffices to deduce a 
contradiction under the following assumption. 
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Assumption 2.2. /5_ < < inf p(X). 

Since iid p{X) > 0, the map / does not admit a fixed point in tt^^{X). The 
overall strategy of the proof is to modify the homeomorphism / away from X to a 
new one g without creating fixed points in A such that the restrictions of g to the 
lifts of the both boundary circles TT~^{d±A) are nontrivial rigid translations by the 
same translation number. Then by glueing the two boundary circles we obtain a 
torus and a homeomorphism on T^. Now we can apply Theorem 12. II to the lift 
of the homeomorphism to the universal covering space. This yields a topological 
foliation on T^, which has long been well understood. The proof will be done by 
analyzing the foliation. We first prepare a lemma which is necessary for the desired 
modification. We do not presume Assumption 12 . 2 1 in the following. 

Lemma 2.3. Assume f does not admit a fixed point in Tr^^(X). Then the prime 
end rotation number p± is nonzero. 

Proof: Consider the mapping / — Id defined on A. Since it is T-invariant, 
it yields a mapping from A, still denoted by the same letter. Then since there is 
no fixed point of / in X, we have (/ — ld){X) C \ {0}. Therefore there is an 
annular open neighbourhood V oi X for which we get a mapping 

/-Id : y ^R2\{0}. 

Clearly for any positively oriented essential simple closed curve 7 in V, the degree 
of the map 

/-Id:7^R2\{0} 
must be the same. If the curve 7 is contained in U±, then the degree can be studied 
by considering the map f± defined on the lift U± of the prime end compactification 
U±. If the prime end rotation number p± is nonzero, the degree is clearly 0. Notice 
that our definition of the degree differs from the usual definition of the index. 

To analyze the case p± = 0, we need the following form of the Cartwright- 
Littlewood theorem |CL] . 

Theorem 2.4. If p+ = and z/Fix(/) nTT~^{X) = 0, then the map /+ on dooU+ 
is Morse Smale and the attractors (resp. repellors) of f+\d^u+ are attractors (resp. 
repellors ) of the whole map /+ . 

This is slightly stronger than the usual version in which it is assumed that 
Fix(/) nX = 0. However the proof works as well under the assumption of Theorem 
[2:41 See e. g. Sect. 3 of iMN, . 

Let us complete the proof of Lemma 12.31 Theorem 12.41 enables us to compute 
the degree of the curve 5 in U± when /5± = 0. The degree is n if 5 C U- and —n 
if (5 C U+, where n is the number of the attractors. Since the degree must be the 
same in [/_ and [/+, the conclusion follows. □ 

Now we have p^ < Q and p+ ^ Q hy Assumption 12.21 and Lemma [2.31 Let us 
start the modification of /. 

Lemma 2.5. Under A ssumption \2.2i there exists a homeomorphism g of A such 
that 

(1) g ^ f in some neighbourhood of X , 

(2) q does not admit a fixed point in A, where g is the lift of g such that g — f on 
n-\X), 
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(3) g is a negative rigid translation by the same translation number on n~^{d±A), 
and 

(4) p- o 9- — p- < — c on U- for some positive number c. 

Proof: The modification in U- will be done in the following way. We identify 
U- with X [—1,0] by the homeomorphism and the universal covering space 
U- withMx [— 1, 0]. Thusp- is just the projection onto the first factor; p-{^, rf) = ^. 
Since p- < 0, the lift 

/_ : R X [-1,0] ^ R X [-1,0] 

of /_ satisfies that p- o /_(^, 0) < ^ — 2c for some c > 0. Therefore changing the 
coordinates of [—1,0] if necessary, one may assume that p_ o < ^ — c if 

(^,7?) G R X [-1/2,0]. Define a homeomorphism h oi x [-1,0] by 

K^,V) = (^ + </?(?7)modl,?7), 

[—1,0] — > (—00,0] is a continuous function such that (^([—1/2,0]) = 

^(r/) < -sup{(p_o/_ -p_)($,77) \ ^eS'}-c. 
f oh. Then its lift g- satisfies 

P- ° .9- ^ P- < 

on J/- = R X [— 1, 0]. Clearly condition (3) for 'K~^(d-A) can be established by a 
further obvious modification. 

Now to modify / in 11+, wc do the same thing as in C/_. If the prime end 
rotation number /9+ is negative, then with an auxiliary modification we are done. 
If it is positive insert a time one map of the Reeb flow. □ 

Consider the torus which is obtained from A by glueing the two bound- 
ary curves d-A and 9+ A. Then the condition (3) above shows that g induces a 
homeomorphism of T^, again denoted by g. The universal cover of is and 
A = R X [—1,1] is a subset of M^. The lift g : A —> A can be extended uniquely to a 
lift ^ : R^ — > R^ of g : ^ T^. The covering transformation group F is isomorphic 
to Z^, generated by the horizontal translation T and the vertical translation by 2, 
denoted by S. Since ^ is a Brouwer homeomorphism which commutes with T, there 
is a F- invariant oriented foliation on R^ whose leaves are Brouwer lines for g. This 
yields an oriented foliation J-' on the torus T^. The proof is divided into several 
cases according to the topological type of the foliation J^. We are going to deduce 
a contradiction in each case. But before going into detail we need another lemma. 

Lemma 2.6. For any C > there is n> such that pog^—p>C on X. 

Proof: If not, there would be a point x„ e X for any n > such that 

n-1 

{pog'^ -p){xn) = ^{pog-p) [g^ (a;„ ) ) < C 

3=0 

for some C > 0, and the averages of Dirac masses 

^ n— 1 

n ^-^ 

j=0 



where : 
and 

Define g = 
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would satisfy ^n{p°9 ^ p) < C jn. Therefore an accumulation point of [in would 
have the property that o g — p) < 0, contradicting the assumption inf > 
0. □ 

Case 1. The foliation J- does not admit a compact leaf. Then J- is conjugate 
either to a linear foliation or to a Denjoy foliation, both of irrational slope. The lift 
of to the open annulus M."^/ (T) is conjugate to a foliation by vertical lines. The 
space of leaves of T is homeomorphic to and there is a projection from R^/ (T) 
to along the leaves of the foliation. This lifts to a projection g : — > M. 

Now q restricted to A is within a bounded error of the first factor projection 
p : ^ M that we have used for the definition of the rotation set p{X). In fact 
both p and q are lifts of degree one maps from M^/(r) to and their difference 
is bounded on the preimage A = Tr~^(A) of a compact subset A. Thus Lemma 12.61 
shows that q o ^"(a;) — > cx) (rt — !■ oo) for x G Tr~^{X). That is, the foliation is 
oriented upward. But this shows that qog(x) > q{x) even for a point x £ Tr^^{d-A). 
On the other hand by condition (3) of Lemma 12. 5[ g is a negative translation on 
TT~^{d-A). A contradiction. 

Case 2.1. The foliation J- admits a compact leaf L of nonzero slope and does 
not admit a Reeb component. In this case the lifted foliation is also conjugate to 
the vertical foliation and the argument of Case 1 applies. 

Case 2.2. The foliation J- admits a Reeb component R of nonzero slope. The 
Brouwer property of leaves implies that g{R) C Int(i?) or g~^{R) C Int(i?). That 
is, a point of the boundary of R is wandering under g. Therefore d-A, consisting 
of nonwandering points of g according to (3) of Lemma 12.51 cannot intersect the 
boundary of R, which is however impossible since the slope of R is nonzero. 

Case 2.3. The foliation T admits a compact leaf of slope 0. Hereafter we 
only consider the dynamics and the foliation on the open annulus M^/(r). Recall 
that A is a subset of M^/ (T), and the homeomorphism g on A is extended to the 
whole IR^/(T), again denoted by in such a way that g commutes with the vertical 
translation S, while the foliation is denoted by as before. 

Now the foliation T yields a partition V of the open annulus M^/(T) into 
compact leaves, interiors of Reeb components and foliated /-bundles. The set V is 
totally ordered by the height. The minimal element which intersects X cannot be 
a compact leaf by the Brouwer line property. Let R be the closure of the minimal 
element. Thus R is either a Reeb component or a foliated /-bundle such that 
Int(i?) n X ^ and d-R. fl X = 0, where d-R is the lower boundary curve of R. 

Assume for a while that d^R is oriented from the right to the left. Thus the 
homeomorphism g carries d^R into the upper complement of d-R. 

Case 2.3.1 R is a Reeb component. First notice that g{R) C Int/? and that 
the interior leaves of R are oriented upwards by the assumption inf p{X) > and 
the fact that g(X n R) C X D R. Choose a simple arc 

a : [0,1] ^ Tr'\R) 

such that a{Q) e n-^{d-R), a{l) = g{a{0)), and a((0,l)) C Int(7r-i(i?)) \ 
g(Tr~^(R)). Since g~^{Tr{a)) is below Int/?, g^^{a), and hence a, is contained 
in 7r-i(C/_). 

Concatenating nonnegative iterates of a, we obtain a simple path 7 : [0, 00) — > 
7r"i(i? n U-) such that g o j{t) = j{t + 1) for any i > 0. Let g : TT^^(lnt{R)) R 
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be the lift of the projection along the leaves. Since 7([l,oo)) is contained in the 
lift of a compact subset g{R) C Int(i?) and the leaves in Int(i?) is oriented upward, 
we have q o ^{t) ^ oo as t ^ oo. We also have p o ^{t) — > oo because q is within 
bounded error of p on 7([1, oo)). 

On the other hand by condition (4) of Lemma |2.5[ we have p o j(t) — — oo as 
t — > oo. In particular the curve 7 is proper both in A and in pointing toward the 
opposite direction. By joining the point 7(0) to an appropriate point in 7r~^(9_A), 
we obtain a simple curve S in 7r^^(J7_) starting at a point on Tr^^{d-A) which 
extends 7. 

Notice that there is a point of tt~^{X) on the left of a proper oriented curve 
S in A, because the map p is bounded from below on S and a high iterate of 
carries a point in tt^^{X) beyond that bound. (There might be a point of Tr^^{X) 
on the right of S however.) 

Let a; be a point in TT~^{d-A) left to the initial point of S. Then there is a 
simple path /3 : [0, 00) — J> tt~^{U-) such that /3(0) = x, limt^oo f3{t) G tt~^{X), and 
/? is disjoint from 6. The path /3, extendable in Tr~^{A) is also extendable in [/_, 
the lift of the prime end compactification. (See e. g. Lemma 2.5 of jMNj .) This 
implies that /? defines a simple path in joining x to a prime end in tt~^ (daoU -) 
without intersecting 5, which is impossible since 'K~^(dooU-) is contained in the 
right side of the proper path 5 in [/_ since P-S{t) — !> —00, while x is on the left 
side. A contradiction. 

Case 2.3.2 R is a foliated I-bundle. Thus the upper boundary curve of 
R is also oriented from the right to the left, and its image by g lies on the upper 
complement of R. The interior leaves of R are oriented upward. 

Recall that the boundary component d-A consisting of nonwandering points 
cannot intersect a compact leaf. Moreover d-A lies in a Reeb component or a 
foliated /-bundle whose interior leaves are oriented downward since pg^{x) —00 
as n ^> 00 for a: G 'K~^{d-A). Let C be the annulus in R^/(T) bounded by d-A 
and 9+-R, the upper boundary curve of R. Notice that Int(C) contains d-R. 

Case 2.3.2.1 The intersection X CiC has a component which separates d-A 
from One can derive a contradiction by the same argument as in Case 2.3.1, 

since the like defined path 7 cannot evade R. 

Case 2.3.2.2 There is a simple path in U- joining a point in d-A with a point 
in d+R. Notice first of all that g~^{C) C C. Let y be the family of the connected 
components of ^^^{X n C). Then any element F G 3^ is compact, and intersects 
'K~^(d+R) since otherwise Y would be a connected component of ti~^{X) itself. 

Choose a simple curve 7 : [0, 1] — J> 7r~^(C) such that 

(1) 7(0)evr-i(a_A), 

(2) 7(1) e 7r-i(XnC), and 

(3) 7([0,l))C7r-i({/-nC). 

Let Y be an element of 3^ which contains 7(1). Then there are two unbounded 
connected components of the complement 7r^^(C) \ (F U 7), one L{Y U 7) on the 
left, and the other R{Y U 7) on the right. 

Notice that for any n > 0, ^^"7 is a path in C, and that p.9^"(7(l)) —00 
and P5~"(7(0)) — 00 as n — >■ 00. That is, for any large ^^"(7(1)) e L{Y U 7) 
and ^""(7(0)) € R{Y U 7), showing that .g~"(7) intersects 7. On the other hand 
in {/_, 7 defines a curve from a point in 7r^^(9_A) to a prime end in 7r^^(i9ooC^-)- 
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But by condition (4) of Lemma [2.51 7 cannot intersect g~"(7) for any large n. A 
contradiction. 

Finally the case where d-R is oriented from the left to the right can be dealt 
with similarly by reversing the time. This completes the proof of Theorem [1] 
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